Dynamic virtual element positioning in an augmented reality environment

ABSTRACT

Systems, methods, and devices relating to dynamic virtual element positioning in an augmented reality (AR) environment are described herein. In a method, a viewport may be output by a computing device and may comprise a virtual element and at least a portion of a scene captured by a camera associated with the computing device. A movement of the at least a portion of the scene in the viewport may be determined. An offset focal point along at least a portion of an axis of the virtual element may be determined that is the closest to a viewport focal point at the vertical and horizontal center of the viewport. The virtual element may be positioned in the viewport based on the movement of the at least a portion of the scene in the viewport and a distance between the offset focal point and the viewport focal point.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/819,848, filed Mar. 16, 2020, which is hereby incorporated byreference in its entirety.

BACKGROUND

In an augmented reality (AR) environment, one or more computer-generatedvirtual elements may be displayed along with one or more real (i.e.,so-called “real world”) elements. For example, a real-time image orvideo of a surrounding environment may be shown on a computer screendisplay with one or more overlaying virtual elements. Such virtualelements may provide complementary information relating to theenvironment or generally enhance a user's perception and engagement withthe environment. Conversely, the real-time image or video of thesurrounding environment may additionally or alternatively enhance auser's engagement with the virtual elements shown on the display. As oneexample, a mobile device, such as a smart phone, may be used to realizean AR environment. A real-time video of the surrounding real-worldenvironment may be captured by a camera of the mobile device and thatvideo scene may be shown on the mobile device's screen display. Virtualelements associated with the real-world environment may be overlaid onportions of the displayed video. As the user moves and/or changes thedirection of the camera, the virtual elements may be modifiedaccordingly. For example, virtual elements may be repositioned, removed,or added. Further, the appearance or content of virtual elements may beadjusted.

Challenges remain in presenting a virtual element in an AR environmentin a manner most beneficial to a particular application. For example, avirtual element may be configured to move within a viewport of the ARenvironment based on movements of the associated camera. User engagementand interaction with the virtual element may be frustrated if thevirtual element is allowed to move too far or even completely out ofview due to certain movements of the camera. Additional difficulties mayarise if movement of the virtual element is indiscriminately limitedalong all axes of the virtual element, particularly when the virtualelement does not have equal vertical and horizontal dimensions.

SUMMARY

Systems, methods, and devices relating to dynamic virtual elementpositioning in an augmented reality (AR) environment are described.

A method may comprise outputting, by a computing device, a viewport(e.g., an AR viewport) that comprises a virtual element and a least aportion of a scene captured by a camera associated with the computingdevice. The virtual element may be elongated. The computing device maycomprise a smart phone or other mobile device that is configured withsaid camera, for example. A movement of the scene in the viewport may bedetermined. Such movement of the scene in the viewport may correspondwith movement of the computing device or camera, for example. An offsetfocal point along at least a portion of an axis (e.g., a longitudinalaxis) of the virtual element may be determined. The determined offsetfocal point may be the point along the at least a portion of the axisthat is closest to a viewport focal point of the viewport. The viewportfocal point may be at the vertical and horizontal center of the viewportand may correspond to the real-world space directly in front of thecamera.

The virtual element may be positioned in the viewport based on themovement of the scene in the viewport and a distance between the offsetfocal point and the viewport focal point. Movement of the scene in theviewport may cause an opposite movement of the virtual element in theviewport, although such movement of the virtual element may berestricted or subject to corrective movement based on the distancebetween the offset focal point and the viewport focal point. The methodand other techniques described herein may allow a user to more easilyview one or more end portions of the virtual element (e.g., the topand/or bottom portions if vertically elongated or the right and/or leftportions if horizontally elongated) while the positioning or movement ofthe virtual element corresponding to its non-elongated dimension may bemore strictly limited or corrected.

As an example, the virtual element may move in the viewport from aninitial position (e.g., the position prior to the movement of the scenein the viewport) to a second position subsequent to the movement of thescene in the viewport. The second position of the virtual element may besuch that the distance between the offset focal point and the viewportfocal point is less than or equal to a predetermined distance threshold.The initial position of the virtual element may be such that the offsetfocal point is less than or equal to the distance threshold or may besuch that the offset focal point is greater than the distance threshold.In the former case, for example, the virtual element may be consideredas moving from an initial acceptable position relative to the viewportto a second position that is also within acceptable limits relative tothe viewport. In the latter case, for example, the virtual element maybe considered as moving from an initial position that is beyondacceptable limits relative to the viewport to a second position that iswithin acceptable limits relative to the viewport. In this latter case,for example, the virtual element or portions thereof may be consideredas being initially too far out of the viewport or too near a border ofthe viewport.

As another example, the position of the virtual element in the viewportmay remain constant during the movement of the scene in the viewport.This may occur, for instance, when the distance between the viewportfocal point and the offset focal point before the movement of the scenein the viewport may be equal to the distance threshold. That is, theposition of the virtual element may be at the cusp of acceptable limits,at least with respect to the movement of the virtual element that maytypically result from said movement of the scene in the viewport. Ratherthan allow a movement of the virtual element that would cause thedistance between the offset focal point and the viewport focal point toexceed the distance threshold, the position of the virtual element maybe instead held constant over the movement of the scene in the viewport.This may appear to a user as if the virtual element is being pulledalong with the scene in the viewport and/or camera as it moves.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to limitations that solve anyor all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate embodiments and together with thedescription, serve to explain the principles of the systems, methods,and devices:

FIGS. 1A-B are diagrams of an example computing device.

FIGS. 2A-E are diagrams of an example computing device under variousstages of movement.

FIGS. 3A-E are diagrams of an example computing device under variousstages of movement.

FIGS. 4A-E are diagrams of an example computing device under variousstages of movement.

FIGS. 5A-D are diagrams of an example computing device under variousstages of movement.

FIG. 6 is a flow diagram of an example method.

FIG. 7 is a block diagram of an example computing device.

Aspects of the disclosure will now be described in detail with referenceto the drawings, wherein like reference numbers refer to like elementsthroughout, unless specified otherwise.

DETAILED DESCRIPTION

Systems, methods, and devices relating to dynamic virtual elementpositioning in an augmented reality (AR) environment are described. AnAR environment may comprise a viewport with a virtual element thatoverlays at least a portion of a captured video scene. The viewport maybe output by a computing device, such as a smart phone configured with acamera that captures the video scene displayed in the viewport. Thevirtual element may be elongated, such as a vertically-oriented virtualremote control. Subject to one or more of the various conditionsdescribed herein, the device (e.g., the camera associated with thedevice) may be moved to cause an opposite movement of the virtualelement in the viewport. This may result in the visual impression thatthe virtual element is present in the real-world space captured by thecamera. As noted, however, the movement or positioning of the virtualelement in the viewport may be restricted or subject to correction ifthe virtual element is moved or would be moved to a position that isbeyond acceptable limits in relation to the viewport (e.g., a focalpoint at the center of the viewport). According to the techniquesdescribed herein, movement of the virtual element corresponding to itselongated dimension may be less restricted than movement of the virtualelement corresponding to its non-elongated dimension. This may enable auser to more easily view features of the virtual element at the endportions of the virtual element, such as buttons or other controls atthe top or bottom of a virtual remote control.

As an example, the viewport may comprise a vertically-elongated virtualelement. An upward movement of the device (which may cause acorresponding upward movement of the scene in the viewport) maytypically cause a downward movement of the virtual element in theviewport and vice versa. Similarly, a leftward movement of the device(which may cause a corresponding leftward movement of the scene in theviewport) may typically cause a rightward movement of the virtualelement in the viewport and vice versa. Similar concepts same may applyto diagonal movements, as well as zoom-in/zoom-out movements. Based on adetermined movement or predicted movement of the device (e.g., movementof the scene in the viewport), an offset focal point along thelongitudinal axis of the virtual element may be determined that isclosest to the focal point at the center of the viewport. Thepositioning of the offset focal point on the longitudinal axis of thevirtual element may be regarded in some sense as the offset focal pointsliding along the longitudinal axis according to the relative positionof the viewport focal point.

The virtual element may be positioned or moved based on the movement ofthe device and the distance between the offset focal point and theviewport focal point. For example, if the offset focal point is too faraway from the viewport focal point, the virtual element may be moved sothat the position of the offset focal point (and the virtual elementgenerally) is within acceptable limits. As another example, if theposition of the offset focal point is within acceptable limits relativeto the viewport focal point, the virtual element may move according tothe movement of the device if the resulting position of the offset focalpoint would be within acceptable limits relative to the viewport focalpoint. Yet if the resulting position of the offset focal point wouldexceed acceptable limits, the movement of the virtual element may berestricted, at least in part, so that the position of the offset focalpoint remains within acceptable limits. As another example, if theposition of the offset focal point is already at the acceptable limitrelative to the viewport focal point, the position of the virtualelement may remain constant such that the virtual element may be seen asbeing pulled along with the movement of the device.

Although described primarily in the context of augmented reality, thedisclosed techniques may be similarly applied in a virtual realityenvironment. For example, the captured video scene shown in the viewportmay be replaced by computer-generated elements in a virtual realityenvironment. These computer-generated elements may be treatedfunctionally in the same manner as if they were a captured video scene.The techniques described herein may be also applied to mixed realityenvironments and other immersion technologies.

FIGS. 1A and 1B illustrate the front and back, respectively, of a device102 configured to realize an AR environment. The device 102 is shownhere as a smart phone, but may comprise various other types of computingdevices. For example, the device 102 may comprise a mobile device, suchas the aforementioned smart phone, a tablet computer, or a portablegaming device. The device 102 may comprise a wearable computing device,such as a head-mounted display or smart glasses. The device 102 maycomprise a virtual retinal display or a contact lens configured todisplay computer-generated elements.

The device 102 may comprise a screen display 104 on the device's 102front and a camera 106 on the device's 102 back. The camera 106 may beconfigured to capture still images and video images. The device 102 maybe additionally or alternatively configured with a camera on its front(not shown). The screen display 104 may comprise an LED or OLED screendisplay. The screen display 104 may comprise a see-through display. Thescreen display 104 may comprise a touchscreen configured to receive userinputs. The device 102 may comprise a gyroscope (e.g., a vibratingstructure microelectromechanical systems (MEMS) gyroscope), anaccelerometer, or other type of movement sensor via which the movementand orientation of the device 102 may be determined.

The screen display 104 may output a viewport 108 configured to display avideo scene captured by the camera 106. The viewport 108 may display thevideo from the camera 106 in real-time or near real-time. The viewport108 may display video from a video source other than the camera 106,such as an external video source. The viewport 108 shown in FIG. 1Acovers the entirety of the screen display 104 (e.g., a full-screenmode), although the disclosure is not so-limited. For example, theviewport 108 may cover only a portion of the screen display 104. Theviewport 108 may be shown in a screen, a window, or a user interface ofan application executing on the device 102, for example.

The viewport 108 may display one or more virtual elements 110. Thevirtual element 110 may be computer-generated by the device 102 andoverlay, at least in part, the captured video scene, including variousreal-world objects in the video. For example, the virtual element 110overlays an entertainment console 112 a and a television 112 b in thevideo scene. The virtual element 110 in FIG. 1A is shown as a remotecontrol for a television, set-top box, or other form of video output ordisplay device. The disclosure is not so-limited, however, and thevirtual element 110 may comprise any sort of object, real or abstract.The virtual element 110 may be partially transparent, in which case theportions of the video scene overlaid by the virtual element 110 may bepartially visible.

The virtual element 110 may be elongated. For example, one dimension ofthe virtual element 110 may be greater than a second dimension of thevirtual element 110 that is perpendicular to the first dimension. Thevirtual element 110 may be elongated along a longitudinal axis of thevirtual element 110. The virtual element 110 may be elongated vertically(e.g., near-vertically) in the viewport 108, as is shown in FIG. 1A.Vertical in this context may refer to the up-down dimension in FIGS. 1Aand 1B, as well as in FIGS. 2A-E through FIGS. 5A-D. The device 102, thescreen display 104, and the viewport 108, as shown in FIG. 1A, may beconsidered as vertically elongated as well. The virtual element 110, thedevice 102, the screen display 104, and/or the viewport 108 may beelongated horizontally or near-horizontally. Horizontal in this contextmay refer to the side-to-side dimension in FIGS. 1A and 1B, as well asin FIGS. 2A-E through FIGS. 5A-D.

The virtual element 110 may be presented as a two-dimensional object. Orthe virtual element 110 may be presented as a three-dimensional object,or with at least some three-dimensional aspects. For example, thevirtual element 110 may enlarge or shrink (e.g., zoom in or zoom out)according to movement of the camera 106, such as movement away from theuser's body or toward the user's body, respectively (or vice versadepending on the particular configuration).

The virtual element 110 may be used for various informational oreducational purposes. For example, the virtual element 110 may provideinformation relating to the real-world environment and/or objects shownin the viewport 108. For example, the virtual element 110 may displayinformation relating to a real-world object in the viewport 108 that thevirtual element 110 overlays or points to. The user may position thedevice 102 so that the virtual element 110 overlays or points to theobject that the user seeks more information on.

Additionally or alternatively, the virtual element 110 may be used toprovide information or an educational experience relating to areal-world object represented by the virtual element 110. For example, auser may indicate (e.g., via a touch input) a feature of the virtualelement 110 to receive additional information about that feature (e.g.,the corresponding feature of a real-world object represented by thevirtual element 110). The information may be provided via audio outputby the device 102, such as a voice description or explanation of anindicated feature and its functions. The information relating to anindicated feature of the virtual element 110 may also be provided via avideo output by the device 102, such as a video providing ademonstration of the feature and associated functions. An animated orcomputer-generated demonstration of an indicated feature of the virtualelement 110 and associated functions may also be shown in the viewport108. Such animation or computer-generated elements may overlay at leasta portion of the captured video scene shown in the viewport 108.

The virtual element movement control techniques described herein mayenhance the information or educational experience afforded by thevirtual element 110. For example, a user may receive the information orengage in the educational experience relating to the virtual element 110in the same or similar environment that the user may actually use areal-world analogue of the virtual element 110, thus providing contextfor this information. For instance, as shown in FIG. 1A, a user mayinteract with the virtual remote control while in the user's actualviewing environment.

A user may indicate a feature of the virtual element 110 via a touchinput to the device 102, such as a touch input to the feature that theuser wishes to receive information on. In the virtual remote controlshown in FIG. 1A, a user may provide a touch input to a button orformation of buttons of the virtual remote control to receiveinformation on the functions of the button(s) on an analogue real-worldremote control.

A feature of the virtual remote control or other type of virtual element110 may be additionally or alternatively indicated according to therelative positioning of that feature within the viewport 108 (e.g., atthe center of the viewport 108). For example, a user may indicate afeature of the virtual element 110 by causing that feature to bepositioned at the center (or other designated area) of the viewport 108.In the virtual remote control example shown in FIG. 1A, a user may havepositioned the circular formation of buttons 114 at the approximatecenter of the viewport 108 to indicate that he or she would likeadditional information relating to the circular formation of buttons114. The indicated feature of the virtual element 110 may be highlighted(or otherwise visually differentiated from the remainder of the virtualelement 110) to reflect that this portion is indeed “indicated.” A textelement (not shown) in the viewport 108 may additionally oralternatively reflect the indicated feature.

The virtual element 110 may move within the viewport 108 based onmovement of the device 102, the camera 106, and/or the scene in theviewport 108. Unless clearly indicated expressly or by context,“movement” as used in this disclosure shall refer to left-right/up-downmovements of the device 102, the camera 106, and/or the scene in theviewport 108, towards-and-away movements of the device 102, the camera106, and/or the scene in the viewport 108, left-right/up-down rotationsof the device 102, the camera 106, and/or the scene in the viewport 108,clockwise/counter-clockwise rotations of the device 102, the camera 106,and/or the scene in the viewport 108, or any combination thereof.

As an example, starting at the position shown in FIG. 1A, the virtualelement 110 may shift in the viewport 108 in a horizontal directionbased on a horizontal movement of the device 102 (and/or the camera 106and/or the scene in the viewport 108) in the opposite direction. Thatis, the virtual element 110 may shift to the left based on a rightwardmovement of the device 102. Or, conversely, the virtual element 110 mayshift to the right based on a leftward movement of the device 102. Thismay simulate or give the visual impression of the virtual element 110remaining in place in the surrounding real-world environment. Yet, inaccordance with the virtual element movement control techniquesdescribed herein, if a horizontal movement of the device 102 continuestoo far in that particular direction, the associated movement of thevirtual element 110 in the opposite direction may be restricted orcorrected for. For example, rather than moving the virtual element 110out of the viewport 108 so as to strictly maintain the visual impressionthat the virtual element 110 remains in place within the real-worldenvironment, the horizontal movement of the virtual element 110 may belimited such that the virtual element 110 remains at least in part(e.g., mostly or completely) within the viewport 108. Once the device102 is no longer being moved in that particular direction, the virtualelement 110 may move back towards a middle position in the viewport 108.In other words, the virtual element 110 may move to “catch up” to thedevice 102. In this manner, the virtual element 110 may generally remainvisible to the user in the viewport 108 while also giving a visualimpression that the virtual element 110 exists in the real world. Insome aspects, the horizontal location of the virtual element 110 in theviewport 108 may remain fixed regardless of any horizontal movement ofthe device 102.

Some of the same or similar concepts may be applied to verticalmovements of the device 102 in the real-world space and associatedvertical movements of the virtual element 110 in the viewport 108. Forexample, an upward movement of the device 102 (and/or the camera 106and/or the scene in the viewport 108) may cause the virtual element 110to shift downward in the viewport 108. Conversely, a downward movementof the device 102 may cause the virtual element 110 to shift upward inthe viewport 108. Like with horizontal movements, this may simulate orgive the visual impression that the virtual element 110 is actuallypresent in the real-world environment. However, due to the elongatedvertical dimensions of the virtual element 110, the methods applied tocontrol the vertical movements of the virtual element 110 may differ insome aspects from those used to control the movements of the virtualelement 110 in the horizontal directions.

For example, a user may wish to view features of the virtual element 110in the virtual element's 110 top portion, such as the MENU button, theGUIDE button, or the circular formation of buttons 114 of the virtualremote control shown in FIG. 1A. The user may thus shift the device 102upward in the real-world space. In accordance with the virtual elementmovement control techniques described herein, the virtual element 110may move correspondingly downward in the viewport 108. The user maycontinue to move the device 102 upward until the top portion of thevirtual element 110 is at the desired position in the viewport 108(e.g., the approximate center of the viewport 108). Rather than limitthe vertical movement of the virtual element 110 so that the virtualelement 110 remains mostly or completely in the viewport 108, as may bethe case with horizontal movement, a greater portion (e.g., a majorityor other predefined proportion) of the virtual element 110 may beallowed to move outside of the viewport 108. Although, the verticalmovement of the virtual element 110 may still be restricted if movementof the device 102 would cause the virtual element 110 to move too far(e.g., past a threshold) out of the viewport 108. For example, thevertical movement of the virtual element 110 may be restricted if suchmovement would cause the virtual element 110 to move completely oralmost completely out of the viewport 108.

The virtual element movement control techniques described herein may beaccomplished, at least in part, via a dynamic offset focal point for thevirtual element 110 that may “slide” along the longitudinal axis of thevirtual element 110 when determining the location and movements of thevirtual element 110 when the device 102 is moved. An offset focal pointfor the virtual element 110 may be determined as a point along thelongitudinal axis of the virtual element 110 that is closest to thefocal point of the viewport 108 (i.e., the center of the viewport 108).This offset focal point on the longitudinal axis of the virtual element110 may be used as a dynamic point of reference to determine if thevirtual element 110 is or will be moved too far, vertically orhorizontally, out of the viewport 108. This is in contrast to using afixed focal point offset for the virtual element 110 that is centeredboth horizontally and vertically on the virtual element 110, which mayprevent the user from viewing and interacting with the desired verticalportions of the virtual element 110 (e.g., the MENU or GUIDE buttons onthe virtual remote control shown in FIG. 1A).

FIGS. 2A-E illustrate the device 102 as it undergoes various stages ofhorizontal movement and any associated horizontal movements of anelongated virtual element 210 in a viewport 208 output by the screendisplay of the device 102. It will be noted that while FIGS. 2A-E (aswell as FIGS. 3A-E, 4A-E, and 5A-D) are described primarily in terms ofmovements of the device 102, the same or similar concepts may be appliedwhen movements of the scene in the viewport 208 are considered as thebases (at least in part) for the virtual element movement controlinstead of or in addition to the movements of the device 102. Forexample, movement of the device 102 in a particular direction may causea corresponding movement of the scene in the viewport 208. Referringback to FIG. 1A as an example, a rightward movement of the device 102may cause a corresponding rightward movement of the scene in theviewport 108 such that the scene in the viewport 108 may then includethe out-of-view wall or other real-world objects to the right of thetelevision 112 b.

The virtual element 210 may be the same as or similar to the virtualelement 110 of FIG. 1A. For example, the virtual element 210 maycomprise a virtual remote control as seen in FIG. 1A, although anybuttons and other features that may often be present on a remote controlor other type of elongated object are not shown on the virtual element210 for clarity of illustration. Aspects of FIGS. 2A-E are notnecessarily shown to scale or may be visually exaggerated also forpurposes of illustration. Although the virtual element 210, the viewport208, and the device 102 are vertically elongated in this example, aswell as in the examples of FIGS. 3A-E, 4A-E, and 5A-D, similar conceptsmay be applied to a horizontally elongated virtual element, ahorizontally elongated viewport, and/or a horizontally elongated device.

FIG. 2A illustrates the device 102 (and the associated camera, e.g., thecamera 106 in FIG. 1B, although not shown) at rest with the approximatemiddle (vertically) portion of the virtual element 210 shown in theviewport 208. The viewport 208 may comprise a focal point 220 at theviewport's 208 center (e.g., the center of the viewport 208 in both thevertical and horizontal dimensions of the viewport 208). The focal point220 may correspond to the real-world space that is directly in front ofthe device's 102 camera. For example, an object shown in the viewport208 at the focal point 220 may be the real-world object directly infront of the camera. Or put another way, a real-world object directly infront of the camera may be shown in the viewport 208 at the focal point220 (or at least would be shown at the focal point 220 if the virtualelement 210 happens to overlay said object in the viewport 208). Thefocal point 220 may remain at a constant position in the viewport 208throughout the various steps shown in FIGS. 2A-E regardless of themovements of the device 102 and the virtual element 210 in the viewport208. Although shown in the various figures of the instant application,the focal point 220 may be a hidden element or attribute of the viewport208 rather than being visually represented in the viewport 208 inpractice.

The virtual element 210 may comprise a longitudinal line segment 226that runs along at least a portion of the longitudinal axis of thevirtual element 210. The longitudinal line segment 226 may be centeredwith respect to the horizontal dimension of the virtual element 210. Thelongitudinal line segment 226 may run from the top of the virtualelement 210 to the bottom of the virtual element 210. The entirety ofthe longitudinal line segment 226 may not be seen in FIG. 2A since onlya portion of the virtual element 210 is shown in the viewport 208. Insome aspects, the longitudinal line segment 226 may run over only aportion of the longitudinal axis of the virtual element 210. Forexample, the longitudinal line segment 226 may run to the top edge ofthe virtual element 210 but not to the bottom edge of the virtualelement 210. Like the focal point 220, the longitudinal line segment 226may not typically be shown in the viewport 208 in practice, as indicatedby the dotted lines of the longitudinal line segment 226.

The virtual element 210 may comprise an offset focal point 222 that isvariably positioned on the longitudinal line segment 226. The positionof the offset focal point 222 may be the position on the longitudinalline segment 226 that is closest to the focal point 220 of the viewport208. The position of the offset focal point 222 may be iterativelydetermined during movements of the device 102. Whether the virtualelement 210 has moved or will move too far out of the viewport 208(e.g., too far away from the focal point 220, whether vertically,horizontally, or a combination thereof) may be determined based on therelative positions of the focal point 220 and the offset focal point222. For example, the virtual element 210 may be determined as being toofar out of the viewport 208 based on a threshold distance d (shown inFIG. 2A) from the offset focal point 222. The threshold distance d maydefine the radius of a distance threshold circle 224 with the offsetfocal point 222 at its center. If the focal point 220 is outside of thedistance threshold circle 224 (i.e., the distance between the focalpoint 220 and the offset focal point 222 exceeds the threshold distanced), the virtual element 210 may be judged as moving too far out of theviewport 208, in which case further movement of the virtual element 210out of the viewport 208 may be limited or the virtual element 210 may bemoved closer to the center of the viewport 208. Like the focal point 220and the longitudinal line segment 226, the offset focal point 222 andthe distance threshold circle 224 may not typically be shown in theviewport 208 in practice.

It is noted that the distance threshold circle 224 is used as a visualrepresentation of the threshold distance d between the focal point 220and the offset focal point 222. The distance threshold circle 224 may bealternatively defined with the focal point 220 at its center, withappropriate adjustments to the techniques described herein. Theresultant movement or positioning of the virtual element may be the sameregardless of whether the distance threshold circle 224 is defined withthe offset focal point 222 at its center or with the focal point 220 atits center.

Continuing to FIG. 2B, the device 102 (e.g., the device's 102 cameraand/or the scene in the viewport 208) is moved horizontally to the rightin the real-world space, as indicated by the corresponding movementarrow. Based on the movement of the device 102, the virtual element 210is moved horizontally to the left in the viewport 208, as also indicatedby the corresponding movement arrow. As noted above, this may simulateor give the vision impression that the virtual element 210 is actuallypresent in the real-world space. Here, the focal point 220 is stillwithin the distance threshold circle 224 (i.e., the distance between thefocal point 220 and the offset focal point 222 is less than thethreshold distance d). The virtual element 210 may thus continue to movewithin the viewport 208 commensurate with the movement of the device102.

In FIG. 2C, the device 102 has continued to move horizontally to theright in the real-world space and the virtual element 210 has movedhorizontally to the left in the viewport 208 accordingly. Here, thefocal point 220 is outside of the distance threshold circle 224 (i.e.,the distance between the focal point 220 and the offset focal point 222is greater than the threshold distance d). The movement or position ofthe virtual element 210 outside of the viewport 208 thus may be beyondacceptable limits. The positions of the virtual element 210 and theoffset focal point 222 in FIG. 2C and other figures may be exaggeratedfor purposes of illustration, although not necessarily so. For example,the comparison between the respective positions of the focal point 220and the offset focal point 222, as well as determining the position ofthe focal point 222 along the longitudinal line segment 226, may be doneon a frame-by-frame basis of the viewport 208 output. Thus the movementof the virtual element 210 may be limited or corrected before thevirtual element 210 reaches the position shown in FIG. 2C. Although itis also possible that the virtual element 210 moved that far between twoframes. Additionally or alternatively, a predictive analysis may beperformed to determine if an expected movement of the virtual element210 will cause the focal point 220 to be outside of the distancethreshold circle 224.

Having determined in relation to FIG. 2C that the virtual element 210has moved too far to the left in the viewport 208, FIG. 2D shows acorrective rightward movement of the virtual element 210 in the viewport208. The virtual element 210 may be moved to the right until the focalpoint 220 is, at the least, at or within the distance threshold circle224.

In FIG. 2E, the device 102 is again moved (or continues to be moved) tothe right in the real-world space. Had the focal point 220 not alreadybeen at the limit of the distance threshold circle 224, the virtualelement 210 may move to the left in the viewport 208 accordingly. Yetbecause any further leftward movement of the virtual element 210 wouldbring the focal point 220 outside of the distance threshold circle 224,such movement of the virtual element 210 is restricted. The virtualelement 210 may appear to a user as being pulled along with the furtherrightward movement of the device 102. The movements of the device 102shown in FIGS. 2B-E may represent a single continuous movement of thedevice 102. Or the movement of the device 102 may cease at FIG. 2D andthen begin again at FIG. 2E. In some aspects, the virtual element 210may resettle towards the horizontal middle of the viewport 208 once thedevice 102 ceases movement.

In some aspects, the process of positioning the virtual element 210within the viewport 208 may completely disallow movement of the virtualelement 210 that would cause the focal point 220 to move outside of thedistance threshold circle 224. Rather, when the focal point 220 reachesthe distance threshold circle 224 or is predicted to be moved outside ofthe distance threshold circle 224, the movement of the virtual element210 in that direction may be disallowed and the virtual element 210 maybe pulled along with any further movement of the device 102 in theopposite direction. The same or similar concepts may apply to thevirtual element movement controls described in relation to FIGS. 3A-E,4A-E, and 5A-D and elsewhere in the instant application.

FIGS. 3A-E illustrate the device 102 as it undergoes various stages ofvertical movement in the real-world space and any associated verticalmovements of an elongated virtual element 310 in a viewport 308 outputby the screen display of the device 102. FIGS. 3A-E also show a focalpoint 320 of the viewport 308 and an offset focal point 322 positionedon a longitudinal line segment 326 of the virtual element 310. Athreshold distance d (shown in FIG. 3A) defines a distance thresholdcircle 324 with the offset focal point 322 at its center. The virtualelement 310, the viewport 308, the focal point 320, the offset focalpoint 322, the longitudinal line segment 326, and the distance thresholdcircle 324 may be the same as or similar to, in at least some aspects,the virtual element 210, the viewport 208, the focal point 220, theoffset focal point 222, the longitudinal line segment 226, and thedistance threshold circle 224 of FIGS. 2A-E, respectively. Aspects ofFIGS. 3A-E are not necessarily shown to scale or may be visuallyexaggerated for purposes of illustration.

FIG. 3A shows the device 102 at rest in the real-world space. The topand middle portions of the virtual element 310 are within the viewport308 while the bottom portion of the virtual element 310 is outside ofthe viewport 308. The position of the offset focal point 322 is withinthe distance threshold circle 324 (i.e., the distance between the focalpoint 320 and the offset focal point 322 is less than the thresholddistance d), so the position of the virtual element 310 may beconsidered as within acceptable limits.

In FIG. 3B, the device 102 is moved upward in the real-world space whilethe virtual element 310 is moved downward in the viewport 308accordingly. For example, a user may move the device 102 upward to moreeasily view features of the virtual element 310 at the top portion ofthe virtual element 310. Additionally or alternatively, the user maycause the top portion of the virtual element 310 to move toward themiddle of the virtual element 310 to indicate that the user wishes toreceive information relating to a feature (e.g., a remote controlbutton) at the top portion of the virtual element 310.

Based on the movements of the device 102, a position of the offset focalpoint 322 along the longitudinal line segment 326 may be determined thatis closest to the focal point 320. In this case, the position of theoffset focal point 322 is at the topmost limit of the longitudinal linesegment 326. The focal point 320 is within the distance threshold circle324, as it is defined based on the determined position of the offsetfocal point 322. Thus the relative positions of the focal point 320 andthe offset focal point 322 do not presently cause movement of thevirtual element 310 to be restricted.

In FIG. 3C, the device 102 is moved further upward in the real-worldspace and the virtual element 310 is moved downward in the viewport 308accordingly. A position of the offset focal point 322 along thelongitudinal line segment 326 that is closest to the focal point 320 maybe again determined. Like in FIG. 3B, said position of the offset focalpoint 322 is also at the topmost limit of the longitudinal line segment326. Alternatively, it may not be required at this point tore-determine, per se, the position of the offset focal point 322 sincethe offset focal point 322 was already at its uppermost position on thelongitudinal line segment 326 in FIG. 3B. Yet in contrast to FIG. 3B,the focal point 320 in FIG. 3C is outside of the distance thresholdcircle 324. The virtual element 310 may thus have moved downward beyondan acceptable limit, as it is defined by the threshold distance d. It isnoted that this and other steps relating to FIGS. 3A-E may be performedin a predictive manner. For example, rather than determining that thevirtual element 310 has already moved too far downward, it may bedetermined that a predicted movement of the virtual element 310 maycause the focal point 320 to be outside of the distance threshold circle324 and the movement of the virtual element 310 may be controlledaccordingly.

In FIG. 3D, the virtual element 310 may undergo corrective movement tocause the focal point 320 to be at or within the distance thresholdcircle 324. Here, the virtual element 310 is moved upward in theviewport 308 such that the focal point 320 is at the distance thresholdcircle 324. In this example, the position of the offset focal point 322is at the same position along the longitudinal line segment 326 as inFIG. 3C, although this may not always be the case. For instance, thevirtual element 310 may have also been caused to move horizontallyand/or rotate in the viewport 308 and this may result in the positionalong the longitudinal line segment 326 that is closest to the focalpoint 322 being somewhere other than at the upper limit of thelongitudinal line segment 326. In either case, the step(s) to performthe corrective movement of the virtual element 310 may includedetermining one or more updated positions of the offset focal point 322as the virtual element 310 moves within the viewport 308.

In FIG. 3E, the device 102 is again (or continues to be) moved upward inthe real-world space. Rather than moving the virtual element 310downward in the viewport 308 based on the upward movement of the device102, the downward movement of the virtual element 310 may be restrictedbecause any further movement in that direction would cause the focalpoint 320 to move outside of the distance threshold circle 324. Thevirtual element 310 may appear to be pulled along with the device 102 asthe device 102 continues to be moved upward in the real-world space.

FIGS. 4A-E illustrate the device 102 as it undergoes various stages ofhorizontal and vertical movement in the real-world space and anyassociated horizontal and vertical movements of an elongated virtualelement 410 in a viewport 408 output by the screen display 104 of thedevice 102. FIGS. 4A-E also show a focal point 420 of the viewport 408and an offset focal point 422 positioned on a longitudinal line segment426 of the virtual element 410. A threshold distance d (shown in FIG.4A) defines a distance threshold circle 424 with the offset focal point422 at its center. The virtual element 410, the viewport 408, the focalpoint 420, the offset focal point 422, the longitudinal line segment426, and the distance threshold circle 424 may be the same as or similarto, in at least some aspects, the virtual element 210, the viewport 208,the focal point 220, the offset focal point 222, the longitudinal linesegment 226, and the distance threshold circle 224 of FIGS. 2A-E,respectively. Aspects of FIGS. 4A-E are not necessarily shown to scaleor may be visually exaggerated for purposes of illustration.

FIG. 4A shows the device 102 at rest in the real-world space. At leastthe approximate middle portion of the virtual element 410 is shown inthe viewport 408. The topmost and the bottommost ends of the virtualelement 410 are outside of the viewport 408. The offset focal point 422is at the position on the longitudinal line segment 426 closest to thefocal point 420 and the resultant distance threshold circle 424encompasses the focal point 420.

In FIG. 4B, the device 102 is moved diagonally downward and to the leftin the real-world space, as indicated by the corresponding movementarrow. Based on the movement of the device 102, the virtual element 410is moved diagonally upward and to the right, as also indicated by thecorresponding movement arrow. The user may have moved the device 102 inthis manner to more easily view a feature (e.g., a button of a remotecontrol) at the bottom portion of the virtual element 410 or to indicatethat the user wishes to receive more information about this feature.

A position of the offset focal point 422 on the longitudinal linesegment 426 may be determined based on the movements of the device 102and virtual element 410 shown in FIG. 4B. In this case, the determinedposition of the offset focal point 422 is at the bottom limit of thelongitudinal line segment 426 (i.e., at the bottom edge of the virtualelement 410). With the position of the offset focal point 422 determinedas such, the focal point 420 is within the distance threshold circle424. That is, the distance between the offset focal point 422 and thefocal point 420 does not exceed the threshold distance d. Accordingly,the position of the virtual element 410 with respect to the viewport 408may be considered as within acceptable limits.

In FIG. 4C, the device 102 is moved further downward and to the left inthe real-world space. Based on this movement of the device 102, thevirtual element 410 is moved further upward and to the right in theviewport 408. A position of the offset focal point 422 along thelongitudinal line segment 426 that is closest to the focal point 420 isagain determined. Alternatively, it may not be required at this point tore-determine, per se, the position of the offset focal point 422 sincethe offset focal point 422 was already at its lowermost position on thelongitudinal line segment 326 in FIG. 4B. Yet in contrast to FIG. 4B,the focal point 420 in FIG. 4C is outside of the distance thresholdcircle 424. That is, the distance between the offset focal point 422 andthe focal point 420 exceeds the threshold distance d. The position ofthe virtual element 410 with respect to the viewport 408 may be deemedas exceeding acceptable limits.

FIG. 4D illustrates a corrective movement of the virtual element 410 tobring its position relative to the viewport 408 within acceptablelimits. In this case, the virtual element 410 is moved downward and tothe left in the viewport 408. The corrective movement of the virtualelement 410 may cause the focal point 420 to be at or within thedistance threshold circle 424. The corrective movement of the virtualelement 410 may be based on the position of the offset focal point 422along the longitudinal line segment 426. Such position of the offsetfocal point 422 may be determined one or more times (e.g., multipletimes) over the course of the corrective movement of the virtual element410 from the position shown in FIG. 4C to the position shown in FIG. 4D.

FIG. 4E illustrates a further movement (or a continuation of themovement shown in FIGS. 4B-C) of the device 102 diagonally downward andto the left in the real-world space. Since the focal point 420 isalready at the limit of the distance threshold circle 424, acorresponding diagonal movement of the virtual element 410 upward and tothe right in the viewport 408 may be restricted rather than allowingfurther movements of the virtual element 410 in that direction.

FIGS. 5A-D illustrate the device 102 as it undergoes various stages ofmovements towards or away from the user in the real-world space and anyassociated movements of an elongated virtual element 510 in a viewport508 output by the screen display 104 of the device 102. FIGS. 5A-D alsoshow a focal point 520 of the viewport 508 and an offset focal point 522positioned on a longitudinal line segment 526 of the virtual element510. A threshold distance d (shown in FIG. 5A) defines a distancethreshold circle 524 with the offset focal point 522 at its center. Thevirtual element 510, the viewport 508, the focal point 520, the offsetfocal point 522, the longitudinal line segment 526, and the distancethreshold circle 524 may be the same as or similar to, in at least someaspects, the virtual element 210, the viewport 208, the focal point 220,the offset focal point 222, the longitudinal line segment 226, and thedistance threshold circle 224 of FIGS. 2A-E, respectively. Aspects ofFIGS. 5A-D are not necessarily shown to scale or may be visuallyexaggerated for purposes of illustration.

FIG. 5A shows the device 102 at rest in the real-world space. A bottomportion of the virtual element 510 is visible in the viewport 508, whilean approximate middle portion and a top portion of the virtual element510 are outside of the viewport 508. The position of the offset focalpoint 522 is at the lowermost limit of the longitudinal line segment 526(i.e., the bottom edge of the virtual element 510) and the focal point520 is within the distance threshold circle 524.

In FIG. 5B, the device 102 is moved toward the user in the real-worldspace, as indicated by the corresponding movement arrow. Such movementof the device 102 in the real-world space may be similar to what onewould do if he or she wishes to zoom out from an object, as the objectwould be seen in the viewport 508. If the right/left movements of thedevice 102 may be considered movements along an X axis and theupward/downward movements of the device 102 may be considered movementsalong a Y axis, the movements of the device 102 toward or away from theuser may be considered movements along a Z axis. Based on the movementof the device 102 toward the user, the virtual element 510 may be shownas if having been zoomed out. As such, the virtual element 510 mayshrink in its proportions, effectively causing the bottom edge of thevirtual element 510 to move upward in the viewport 508. The zoom-outmovement of the device 102 represented in FIG. 5B may not triggercorrective movement or movement restrictions for the virtual element 510since the focal point 520 is within the distance threshold circle 524.In some aspects, a movement of the device 102 away from the user, ratherthan toward the user, may cause the zoomed-out appearance of the virtualelement 510 in the viewport 508, but the same concepts may apply withappropriate adjustments.

In FIG. 5C, the device 102 is further moved toward the user in thereal-world space, thus causing the virtual element 510 to further shrinkin its proportions in the viewport 508, which, in effect, also causesthe bottom edge of the virtual element 510 to shift upward. As a result,the focal point 520 of the viewport 508 is outside of the distancethreshold circle 524, as it is defined with the determined offset focalpoint 522 positioned at the bottom-most limit of the longitudinal linesegment 526.

In FIG. 5D, a corrective movement of the virtual element 510 isimplemented to shift the virtual element 510 downward, as indicated bythe movement arrow. The downward movement of the virtual element 510 maycause the focal point 520 to be at or within the distance thresholdcircle 524. In the example shown in FIG. 5D, the proportions of thevirtual element 510 are held constant between FIGS. 5C and 5D.Additionally or alternatively, the proportions of the virtual element510 may be adjusted (e.g., zoomed in) to cause the focal point 520 to beat or within the distance threshold circle 524.

FIG. 6 illustrates a flow diagram of a method 600 for movement controlof a virtual element in a viewport output by a device (a computingdevice, e.g., the device 102 of FIGS. 1A-B). The viewport may comprisean augmented reality (AR) viewport and the virtual element may beelongated. The virtual element may move in the viewport based onmovement of a scene (e.g., a scene captured by a camera associated withthe device) in the viewport. Such movement of the virtual element in theviewport may replicate a user experience as if the object represented bythe virtual element did in fact exist in the real-world space, forexample. Although, under certain conditions, movement of the virtualelement in the viewport may be restricted or corrected for. For example,movement of the virtual element may be restricted if the position of thevirtual element (or portion thereof) with respect to the viewport is orwill be outside of acceptable limits. The method 600 may allow for lessrestrictive movement or positioning of the virtual element in thedirections corresponding to the virtual element's elongated dimensionthan in the directions corresponding to the virtual element'snon-elongated dimension.

The virtual element may comprise the virtual element 110 of FIG. 1A,such as a virtual remote control. The virtual element may additionallyor alternatively comprise one or more of the virtual elements 210, 310,410, 510 of FIGS. 2A-E, FIGS. 3A-E, FIGS. 4A-E, and FIGS. 5A-D,respectively. The viewport may likewise comprise one or more of theviewports 108, 208, 308, 408, 508 of FIGS. 1A-B, FIGS. 2A-E, FIGS. 3A-E,FIGS. 4A-E, and FIGS. 5A-D, respectively. The virtual element may beelongated in a generally vertical dimension of the viewport, as shown inthe figures, or the virtual element may be elongated in a generallyhorizontal dimension of the viewport.

At step 602, the viewport may be output (e.g., caused to be output) on ascreen display of the device. As noted, the viewport may comprise thevirtual element, as well as at least a portion of a scene captured bythe camera associated with the device. The virtual element may overlayat least a portion of the scene in the viewport. The virtual element maybe movable in the viewport, including movement that takes a portion ofthe virtual element outside of the viewport. The movement of the virtualelement in the viewport may be based on movement of the at least aportion of the scene in the viewport. Additionally or alternatively, themovement of the virtual element in the viewport may be based on movementof the device, movement of the camera, movement of the viewport, datafrom a gyroscope of the device, data from an accelerator of the device,and/or data from a movement sensor of the device. For example, as shownin FIG. 2B, a horizontal movement (e.g., to the right) of the at least aportion of the scene in the viewport may cause an opposite horizontalmovement (e.g., to the left) of the virtual element in the viewport. Asanother example, as shown in FIG. 3B, a vertical movement (e.g., upward)of the at least a portion of the scene in the viewport may cause anopposite vertical movement (e.g., downward) of the virtual element inthe viewport. As another example, as shown in FIG. 4B, a diagonalmovement (e.g., a combined movement to the left and downward) of the atleast a portion of the scene in the viewport may cause an oppositediagonal movement (e.g., a combined movement to the right and upward) ofthe virtual element in the viewport. As another example, as shown inFIG. 5B, a zoom-in or zoom-out movement of the at least a portion of thescene in the viewport may cause the virtual element to shrink or expandin the viewport. The zoom-in or zoom-out movement of the at least aportion of the scene in the viewport may be caused by a movement of thedevice and/or camera toward or away from a user (e.g., a Z axismovement).

At step 604, a movement of the at least a portion of the scene in theviewport may be determined. Determining the movement of the at least aportion of the scene in the viewport may be based on at least one of: amovement of the camera, a movement of the device, a movement of theviewport, data from a gyroscope of the device, data from anaccelerometer of the device, or data from a movement sensor of thedevice. The movement may be horizontal, vertical, diagonal, zoom-in,zoom-out, or any combination thereof.

At step 606, an offset focal point along at least a portion of thevirtual element's axis may be determined. The axis may be a longitudinalaxis of the virtual element. The offset focal point may be the pointalong the at least a portion of the longitudinal axis that is closest toa focal point of the viewport (i.e., a viewport focal point). The offsetfocal point may comprise the offset focal point 222, 322, 422, 522 ofFIGS. 2A-E, FIGS. 3A-E, FIGS. 4A-E, and FIGS. 5A-D, respectively. Theoffset focal point may be determined based on the movement of the atleast a portion of the scene in the viewport. For example, the offsetfocal point may be determined following a movement of the at least aportion of the scene in the viewport, including a completed movement ofthe at least a portion of the scene in the viewport or a movement thatis part of an ongoing movement of the at least a portion of the scene inthe viewport. The offset focal point may be a predicted offset focalpoint that is determined based on the movement or a predicated movementof the at least a portion of the scene in the viewport. Additionally oralternatively, the offset focal point or a predicted offset focal pointmay be determined based on a movement or a predicted movement,respectively, of the device and/or the camera.

The focal point of the viewport may be at the vertical and horizontalcenter of the viewport. The focal point may correspond to the real-worldspace that is directly in front of the camera. The focal point maycomprise the focal point 220, 320, 420, 520 of FIGS. 2A-E, FIGS. 3A-E,FIGS. 4A-E, and FIGS. 5A-D, respectively. The offset focal point may bedetermined and/or re-determined one or more times before, during, orafter a movement of the device, whereas the position of the focal pointwithin the viewport may remain constant regardless of the movement ofthe at least a portion of the scene in the viewport (and/or the movementof the device and/or the movement of the camera).

The longitudinal axis may be parallel to the elongated dimension of thevirtual element and at the middle point in the non-elongated dimensionof the virtual element. The at least a portion of the longitudinal axismay comprise a longitudinal line segment, such as the longitudinal linesegment 226, 326, 426, 526 of FIGS. 2A-E, FIGS. 3A-E, FIGS. 4A-E, andFIGS. 5A-D, respectively. The longitudinal line segment may span theentire longitudinal axis of the virtual element (e.g., from the top edgeto the bottom edge of a vertically elongated virtual element) or only aportion of the longitudinal axis. FIGS. 3A-B provide examples in whichthe offset focal point 322 is at the top edge of the virtual element 310along the longitudinal line segment 326.

At step 608, the virtual element may be positioned in the viewport basedon the movement of the at least a portion of the scene in the viewportand a distance between the offset focal point and the focal point of theviewport. Additionally or alternatively, the virtual element may bepositioned in the viewport based on the movement of the camera and/ordevice and a distance between the offset focal point and the focal pointof the viewport. Positioning the virtual element in the viewport maycomprise moving the virtual element in the viewport. The virtual elementmay be positioned in the viewport based on a distance threshold. Forexample, the distance between the offset focal point and the focal pointmay be compared to the distance threshold.

If the distance between the offset focal point and the focal point isless than or equal to (e.g., satisfies) the distance threshold, theposition and/or movement of the virtual element in the viewport maycorrespond to the movement of the at least a portion of the scene in theviewport without additional limitations. FIG. 3B shows an example inwhich the movement of the virtual element 310 corresponds to themovement of the device 102 (e.g., the movement of the at least a portionof the scene in the viewport) and is at that moment unrestricted by thedistance between the offset focal point 322 and the focal point 320because the focal point 320 is within the distance threshold circle 324(i.e., said distance is less than or equal to the threshold distance d).FIGS. 2B, 4B, and 5B provide similar examples associated with horizontalmovement, diagonal movement, and toward/away movement of the device 102(e.g., horizontal movement, diagonal movement, and zoom-in/zoom-outmovement of the at least a portion of the scene in the viewport),respectively.

Yet if the distance between the offset focal point and the focal pointof the viewport is greater than (e.g., does not satisfy) the distancethreshold, the positioning and/or movement of the virtual element may berestricted and/or subject to correction. For example, the virtualelement may be instead moved (or not moved, as the case may be) in theviewport so that the distance between the offset focal point of thevirtual element and the focal point of the viewport is less than orequal to the distance threshold. This may include preventing a movementof the virtual element that would cause the distance between the offsetfocal point and the focal point of the viewport to exceed the distancethreshold. In this case, the virtual element may appear to be pulledalong with the movement of the camera and/or the at least a portion ofthe scene in the viewport rather than moving further out of theviewport. Additionally or alternatively, the virtual element may besubject to corrective movement, such as if the virtual element wasalready too far out of the viewport. In this case, the virtual elementmay be moved so that the distance between the offset focal point and thefocal point of the viewport is equal to or less than the distancethreshold. Here, the virtual element may be seen as catching up to thecamera and/or the at least a portion of the scene in the viewport.

FIGS. 3C-E show an example in which the positioning and/or movement ofthe virtual element 310 that results (or would result) from theindicated movement of the device 102 (e.g., movement of the at least aportion of the scene in the viewport 308) is restricted and/or correctedbecause the focal point 320 in FIG. 3C is outside of the distancethreshold circle 324 (i.e., said distance is greater than the thresholddistance d). In FIG. 3D, the virtual element 310 is subject tocorrective upward movement in the viewport 308 that causes the focalpoint 320 to be at the distance threshold circle 324. In other words,the virtual element 310 in FIG. 3C is determined to be too far out ofthe viewport 308. The virtual element 310 is moved upward in FIG. 3D sothat the position of the virtual element 310 in relation to the viewport308 is within acceptable limits. In FIG. 3E, the device 102 is movedfurther in the indicated direction that initially caused the virtualelement 310 to move or about to be moved out of the viewport 308. Here,the downward movement of the virtual element 310 that would otherwiseoccur based on the upward movement of the device 102 is prevented sothat the focal point 320 remains at the distance threshold circle 324.The virtual element 310 may appear to be pulled along with the device102 as the device 102 is pulled upward. FIGS. 2C-E, 4C-E, and 5C-Dprovide similar examples associated with horizontal movement, diagonalmovement, and toward/away movement of the device 102, respectively.

FIG. 7 depicts a computing device in which the systems, methods, anddevices disclosed herein, or all or some aspects thereof, may beembodied. For example, components such as the device 102 of FIGS. 1A-Bmay be implemented generally in a computing device, such as thecomputing device 700 of FIG. 7. The computing device of FIG. 7 may beall or part of a server, workstation, desktop computer, laptop, tablet,network appliance, PDA, e-reader, digital cellular phone, set top box,or the like, and may be utilized to implement any of the aspects of thesystems, methods, and devices described herein.

The computing device 700 may include a baseboard, or “motherboard,”which is a printed circuit board to which a multitude of components ordevices may be connected by way of a system bus or other electricalcommunication paths. One or more central processing units (CPUs) 704 mayoperate in conjunction with a chipset 706. The CPU(s) 704 may bestandard programmable processors that perform arithmetic and logicaloperations necessary for the operation of the computing device 700.

The CPU(s) 704 may perform the necessary operations by transitioningfrom one discrete physical state to the next through the manipulation ofswitching elements that differentiate between and change these states.Switching elements may generally include electronic circuits thatmaintain one of two binary states, such as flip-flops, and electroniccircuits that provide an output state based on the logical combinationof the states of one or more other switching elements, such as logicgates. These basic switching elements may be combined to create morecomplex logic circuits including registers, adders-subtractors,arithmetic logic units, floating-point units, and the like.

The CPU(s) 704 may be augmented with or replaced by other processingunits, such as GPU(s) 705. The GPU(s) 705 may comprise processing unitsspecialized for but not necessarily limited to highly parallelcomputations, such as graphics and other visualization-relatedprocessing.

A chipset 706 may provide an interface between the CPU(s) 704 and theremainder of the components and devices on the baseboard. The chipset706 may provide an interface to a random access memory (RAM) 708 used asthe main memory in the computing device 700. The chipset 706 may furtherprovide an interface to a computer-readable storage medium, such as aread-only memory (ROM) 720 or non-volatile RAM (NVRAM) (not shown), forstoring basic routines that may help to start up the computing device700 and to transfer information between the various components anddevices. ROM 720 or NVRAM may also store other software componentsnecessary for the operation of the computing device 700 in accordancewith the aspects described herein.

The computing device 700 may operate in a networked environment usinglogical connections to remote computing nodes and computer systemsthrough local area network (LAN) 716. The chipset 706 may includefunctionality for providing network connectivity through a networkinterface controller (NIC) 722, such as a gigabit Ethernet adapter. ANIC 722 may be capable of connecting the computing device 700 to othercomputing nodes over a network 716. It should be appreciated thatmultiple NICs 722 may be present in the computing device 700, connectingthe computing device to other types of networks and remote computersystems.

The computing device 700 may be connected to a mass storage device 728that provides non-volatile storage for the computer. The mass storagedevice 728 may store system programs, application programs, otherprogram modules, and data, which have been described in greater detailherein. The mass storage device 728 may be connected to the computingdevice 700 through a storage controller 724 connected to the chipset706. The mass storage device 728 may consist of one or more physicalstorage units. A storage controller 724 may interface with the physicalstorage units through a serial attached SCSI (SAS) interface, a serialadvanced technology attachment (SATA) interface, a fiber channel (FC)interface, or other type of interface for physically connecting andtransferring data between computers and physical storage units.

The computing device 700 may store data on a mass storage device 728 bytransforming the physical state of the physical storage units to reflectthe information being stored. The specific transformation of a physicalstate may depend on various factors and on different implementations ofthis description. Examples of such factors may include, but are notlimited to, the technology used to implement the physical storage unitsand whether the mass storage device 728 is characterized as primary orsecondary storage and the like.

For example, the computing device 700 may store information to the massstorage device 728 by issuing instructions through a storage controller724 to alter the magnetic characteristics of a particular locationwithin a magnetic disk drive unit, the reflective or refractivecharacteristics of a particular location in an optical storage unit, orthe electrical characteristics of a particular capacitor, transistor, orother discrete component in a solid-state storage unit. Othertransformations of physical media are possible without departing fromthe scope and spirit of the present description, with the foregoingexamples provided only to facilitate this description. The computingdevice 700 may further read information from the mass storage device 728by detecting the physical states or characteristics of one or moreparticular locations within the physical storage units.

In addition to the mass storage device 728 described above, thecomputing device 700 may have access to other computer-readable storagemedia to store and retrieve information, such as program modules, datastructures, or other data. It should be appreciated by those skilled inthe art that computer-readable storage media may be any available mediathat provides for the storage of non-transitory data and that may beaccessed by the computing device 700.

By way of example and not limitation, computer-readable storage mediamay include volatile and non-volatile, transitory computer-readablestorage media and non-transitory computer-readable storage media, andremovable and non-removable media implemented in any method ortechnology. Computer-readable storage media includes, but is not limitedto, RAM, ROM, erasable programmable ROM (“EPROM”), electrically erasableprogrammable ROM (“EEPROM”), flash memory or other solid-state memorytechnology, compact disc ROM (“CD-ROM”), digital versatile disk (“DVD”),high definition DVD (“HD-DVD”), BLU-RAY, or other optical storage,magnetic cassettes, magnetic tape, magnetic disk storage, other magneticstorage devices, or any other medium that may be used to store thedesired information in a non-transitory fashion.

A mass storage device, such as the mass storage device 728 depicted inFIG. 7, may store an operating system utilized to control the operationof the computing device 700. The operating system may comprise a versionof the LINUX operating system. The operating system may comprise aversion of the WINDOWS SERVER operating system from the MICROSOFTCorporation. According to further aspects, the operating system maycomprise a version of the UNIX operating system. Various mobile phoneoperating systems, such as IOS and ANDROID, may also be utilized. Itshould be appreciated that other operating systems may also be utilized.The mass storage device 728 may store other system or applicationprograms and data utilized by the computing device 700.

The mass storage device 728 or other computer-readable storage media mayalso be encoded with computer-executable instructions, which, whenloaded into the computing device 700, transforms the computing devicefrom a general-purpose computing system into a special-purpose computercapable of implementing the aspects described herein. Thesecomputer-executable instructions transform the computing device 700 byspecifying how the CPU(s) 704 transition between states, as describedabove. The computing device 700 may have access to computer-readablestorage media storing computer-executable instructions, which, whenexecuted by the computing device 700, may perform the methods describedherein.

A computing device, such as the computing device 700 depicted in FIG. 7,may also include an input/output controller 732 for receiving andprocessing input from a number of input devices, such as a keyboard, amouse, a touchpad, a touch screen, an electronic stylus, or other typeof input device. Similarly, an input/output controller 732 may provideoutput to a display, such as a computer monitor, a flat-panel display, adigital projector, a printer, a plotter, or other type of output device.It will be appreciated that the computing device 700 may not include allof the components shown in FIG. 7, may include other components that arenot explicitly shown in FIG. 7, or may utilize an architecturecompletely different than that shown in FIG. 7.

As described herein, a computing device may be a physical computingdevice, such as the computing device 700 of FIG. 7. A computing node mayalso include a virtual machine host process and one or more virtualmachine instances. Computer-executable instructions may be executed bythe physical hardware of a computing device indirectly throughinterpretation and/or execution of instructions stored and executed inthe context of a virtual machine.

It is to be understood that the systems, methods, and devices are notlimited to specific methods, specific components, or to particularimplementations. It is also to be understood that the terminology usedherein is for the purpose of describing particular embodiments only andis not intended to be limiting.

As used in the specification and the appended claims, the singular forms“a,” “an,” and “the” include plural referents unless the context clearlydictates otherwise. Ranges may be expressed herein as from “about” oneparticular value, and/or to “about” another particular value. When sucha range is expressed, another embodiment includes from the oneparticular value and/or to the other particular value. Similarly, whenvalues are expressed as approximations, by use of the antecedent“about,” it will be understood that the particular value forms anotherembodiment. It will be further understood that the endpoints of each ofthe ranges are significant both in relation to the other endpoint, andindependently of the other endpoint.

“Optional” or “optionally” means that the subsequently described eventor circumstance may or may not occur, and that the description includesinstances where said event or circumstance occurs and instances where itdoes not.

Throughout the description and claims of this specification, the word“comprise” and variations of the word, such as “comprising” and“comprises,” means “including but not limited to,” and is not intendedto exclude, for example, other components, integers or steps.“Exemplary” means “an example of” and is not intended to convey anindication of a preferred or ideal embodiment. “Such as” is not used ina restrictive sense, but for explanatory purposes.

Components are described that may be used to perform the describedsystems, methods, and devices. When combinations, subsets, interactions,groups, etc., of these components are described, it is understood thatwhile specific references to each of the various individual andcollective combinations and permutations of these may not be explicitlydescribed, each is specifically contemplated and described herein, forall systems, methods, and devices. This applies to all aspects of thisapplication including, but not limited to, operations in describedmethods. Thus, if there are a variety of additional operations that maybe performed it is understood that each of these additional operationsmay be performed with any specific embodiment or combination ofembodiments of the described methods.

As will be appreciated by one skilled in the art, the systems, methods,and devices may take the form of an entirely hardware embodiment, anentirely software embodiment, or an embodiment combining software andhardware aspects. Furthermore, the systems, methods, and devices maytake the form of a computer program product on a computer-readablestorage medium having computer-readable program instructions (e.g.,computer software) embodied in the storage medium. More particularly,the present systems, methods, and devices may take the form ofweb-implemented computer software. Any suitable computer-readablestorage medium may be utilized including hard disks, CD-ROMs, opticalstorage devices, or magnetic storage devices.

Embodiments of the systems, methods, and devices are described abovewith reference to block diagrams and flowchart illustrations of methods,systems, apparatuses and computer program products. It will beunderstood that each block of the block diagrams and flowchartillustrations, and combinations of blocks in the block diagrams andflowchart illustrations, respectively, may be implemented by computerprogram instructions. These computer program instructions may be loadedon a general-purpose computer, special-purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions which execute on the computer or other programmabledata processing apparatus create a means for implementing the functionsspecified in the flowchart block or blocks.

These computer program instructions may also be stored in acomputer-readable memory that may direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including computer-readableinstructions for implementing the function specified in the flowchartblock or blocks. The computer program instructions may also be loadedonto a computer or other programmable data processing apparatus to causea series of operational steps to be performed on the computer or otherprogrammable apparatus to produce a computer-implemented process suchthat the instructions that execute on the computer or other programmableapparatus provide steps for implementing the functions specified in theflowchart block or blocks.

The various features and processes described above may be usedindependently of one another, or may be combined in various ways. Allpossible combinations and sub-combinations are intended to fall withinthe scope of this disclosure. In addition, certain methods or processblocks may be omitted in some implementations. The methods and processesdescribed herein are also not limited to any particular sequence, andthe blocks or states relating thereto may be performed in othersequences that are appropriate. For example, described blocks or statesmay be performed in an order other than that specifically described, ormultiple blocks or states may be combined in a single block or state.The example blocks or states may be performed in serial, in parallel, orin some other manner. Blocks or states may be added to or removed fromthe described example embodiments. The example systems and componentsdescribed herein may be configured differently than described. Forexample, elements may be added to, removed from, or rearranged comparedto the described example embodiments.

It will also be appreciated that various items are illustrated as beingstored in memory or on storage while being used, and that these items orportions thereof may be transferred between memory and other storagedevices for purposes of memory management and data integrity.Alternatively, in other embodiments, some or all of the software modulesand/or systems may execute in memory on another device and communicatewith the illustrated computing systems via inter-computer communication.Furthermore, in some embodiments, some or all of the systems and/ormodules may be implemented or provided in other ways, such as at leastpartially in firmware and/or hardware, including, but not limited to,one or more application-specific integrated circuits (“ASICs”), standardintegrated circuits, controllers (e.g., by executing appropriateinstructions, and including microcontrollers and/or embeddedcontrollers), field-programmable gate arrays (“FPGAs”), complexprogrammable logic devices (“CPLDs”), etc. Some or all of the modules,systems, and data structures may also be stored (e.g., as softwareinstructions or structured data) on a computer-readable medium, such asa hard disk, a memory, a network, or a portable media article to be readby an appropriate device or via an appropriate connection. The systems,modules, and data structures may also be transmitted as generated datasignals (e.g., as part of a carrier wave or other analog or digitalpropagated signal) on a variety of computer-readable transmission media,including wireless-based and wired/cable-based media, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). Suchcomputer program products may also take other forms in otherembodiments. Accordingly, the present invention may be practiced withother computer system configurations.

While the systems, methods, and devices have been described inconnection with preferred embodiments and specific examples, it is notintended that the scope be limited to the particular embodiments setforth, as the embodiments herein are intended in all respects to beillustrative rather than restrictive.

Unless otherwise expressly stated, it is in no way intended that anymethod set forth herein be construed as requiring that its operations beperformed in a specific order. Accordingly, where a method claim doesnot actually recite an order to be followed by its operations or it isnot otherwise specifically stated in the claims or descriptions that theoperations are to be limited to a specific order, it is no way intendedthat an order be inferred, in any respect. This holds for any possiblenon-express basis for interpretation, including: matters of logic withrespect to arrangement of steps or operational flow; plain meaningderived from grammatical organization or punctuation; and the number ortype of embodiments described in the specification.

It will be apparent to those skilled in the art that variousmodifications and variations may be made without departing from thescope or spirit of the present disclosure. Other embodiments will beapparent to those skilled in the art from consideration of thespecification and practices described herein. It is intended that thespecification and example figures be considered as exemplary only, witha true scope and spirit being indicated by the following claims.

What is claimed is:
 1. A method comprising: determining a movementassociated with a viewport; determining a point along an axis of avirtual element that is closest to a focal point of the viewport; andpositioning, based at least on the movement associated with the viewportand a distance between the point along the axis of the virtual elementand the focal point of the viewport, the virtual element in theviewport.
 2. The method of claim 1, wherein the virtual elementcomprises a virtual remote control.
 3. The method of claim 1, furthercomprising: comparing the distance between the point along the axis ofthe virtual element and the focal point of the viewport to a distancethreshold, wherein the positioning the virtual element in the viewportis based on the comparing the distance between the point along the axisof the virtual element and the focal point of the viewport to thedistance threshold.
 4. The method of claim 3, further comprising:determining that the distance between the point along the axis of thevirtual element and the focal point of the viewport exceeds the distancethreshold; and positioning the virtual element in the viewport such thatthe distance between the point along the axis of the virtual element andthe focal point of the viewport is less than or equal to the distancethreshold.
 5. The method of claim 3, further comprising: determiningthat the distance between the point along the axis of the virtualelement and the focal point of the viewport is less than or equal to thedistance threshold; and positioning the virtual element in the viewportsuch that the distance between the point along the axis of the virtualelement and the focal point of the viewport remains less than or equalto the distance threshold.
 6. The method of claim 3, further comprising:determining that the distance between the point along the axis of thevirtual element and the focal point of the viewport is less than orequal to the distance threshold; and maintaining the positioning of thevirtual element in the viewport during the movement associated with theviewport.
 7. The method of claim 1, wherein the movement associated withthe viewport causes an opposite movement of the virtual element in theviewport.
 8. The method of claim 1, wherein the movement associated withthe viewport comprises a predicted movement of at least a portion of ascene in the viewport and the point along the axis of the virtualelement comprises a predicted offset focal point based on the predictedmovement of the at least a portion of the scene in the viewport.
 9. Themethod of claim 1, wherein the virtual element is at least one ofvertically elongated or horizontally elongated.
 10. The method of claim1, wherein the determining the movement associated with the viewport isbased on at least one of: a movement of a camera, a movement of acomputing device, a movement of the viewport, data from a gyroscope ofthe computing device, data from an accelerometer of the computingdevice, or data from a movement sensor of the computing device.
 11. Themethod of claim 10, wherein the computing device comprises at least oneof a mobile device, a smart phone, a tablet computer, a portable gamingdevice, a wearable computing device, a head-mounted display, smartglasses, a virtual retinal display, or a contact lens configured todisplay computer-generated elements.
 12. The method of claim 1, whereinthe axis of the virtual element comprises a longitudinal axis of thevirtual element.
 13. The method of claim 1, wherein the viewportcomprises an augmented reality (AR) viewport.
 14. The method of claim 1,further comprising: indicating, by a user and based on the positioningthe virtual element in the viewport, a feature of the virtual element.15. A device comprising: one or more processors; and memory storinginstructions that, when executed by the one or more processors, causethe device to: determine a movement associated with a viewport;determine a point along an axis of a virtual element that is closest toa focal point of the viewport; and position, based at least on themovement associated with the viewport and a distance between the pointalong the axis of the virtual element and the focal point of theviewport, the virtual element in the viewport.
 16. The device of claim15, wherein the virtual element comprises a virtual remote control. 17.The device of claim 15, wherein the instructions, when executed by theone or more processors, further cause the device to: compare thedistance between the point along the axis of the virtual element and thefocal point of the viewport to a distance threshold, wherein thepositioning the virtual element in the viewport is based on thecomparing the distance between the point along the axis of the virtualelement and the focal point of the viewport to the distance threshold.18. The device of claim 17, wherein the instructions, when executed bythe one or more processors, further cause the device to: determine thatthe distance between the point along the axis of the virtual element andthe focal point of the viewport exceeds the distance threshold; andposition the virtual element in the viewport such that the distancebetween the point along the axis of the virtual element and the focalpoint of the viewport is less than or equal to the distance threshold.19. The device of claim 17, wherein the instructions, when executed bythe one or more processors, further cause the device to: determine thatthe distance between the point along the axis of the virtual element andthe focal point of the viewport is less than or equal to the distancethreshold; and position the virtual element in the viewport such thatthe distance between the point along the axis of the virtual element andthe focal point of the viewport remains less than or equal to thedistance threshold.
 20. The device of claim 17, wherein theinstructions, when executed by the one or more processors, further causethe device to: determine that the distance between the point along theaxis of the virtual element and the focal point of the viewport is lessthan or equal to the distance threshold; and maintain the positioning ofthe virtual element in the viewport during the movement associated withthe viewport.
 21. The device of claim 15, wherein the movementassociated with the viewport causes an opposite movement of the virtualelement in the viewport.
 22. The device of claim 15, wherein themovement associated with the viewport comprises a predicted movement ofat least a portion of a scene in the viewport and the point along theaxis of the virtual element comprises a predicted offset focal pointbased on the predicted movement of the at least a portion of the scenein the viewport.
 23. The device of claim 15, wherein the virtual elementis at least one of vertically elongated or horizontally elongated. 24.The device of claim 15, wherein the determining the movement associatedwith the viewport is based on at least one of: a movement of a camera, amovement of a computing device, a movement of the viewport, data from agyroscope of the computing device, data from an accelerometer of thecomputing device, or data from a movement sensor of the computingdevice.
 25. The device of claim 24, wherein the computing devicecomprises at least one of a mobile device, a smart phone, a tabletcomputer, a portable gaming device, a wearable computing device, ahead-mounted display, smart glasses, a virtual retinal display, or acontact lens configured to display computer-generated elements.
 26. Thedevice of claim 15, wherein the axis of the virtual element comprises alongitudinal axis of the virtual element.
 27. The device of claim 15,wherein the viewport comprises an augmented reality (AR) viewport. 28.The device of claim 15, wherein the instructions, when executed by theone or more processors, further cause the device to: indicate, by a userand based on the positioning the virtual element in the viewport, afeature of the virtual element.
 29. A non-transitory computer-readablemedium storing instructions that, when executed, cause: determining amovement associated with a viewport; determining a point along an axisof a virtual element that is closest to a focal point of the viewport;and positioning, based at least on the movement associated with theviewport and a distance between the point along the axis of the virtualelement and the focal point of the viewport, the virtual element in theviewport.
 30. The non-transitory computer-readable medium of claim 29,wherein the virtual element comprises a virtual remote control.
 31. Thenon-transitory computer-readable medium of claim 29, wherein theinstructions, when executed, further cause: comparing the distancebetween the point along the axis of the virtual element and the focalpoint of the viewport to a distance threshold, wherein the positioningthe virtual element in the viewport is based on the comparing thedistance between the point along the axis of the virtual element and thefocal point of the viewport to the distance threshold.
 32. Thenon-transitory computer-readable medium of claim 31, wherein theinstructions, when executed, further cause: determining that thedistance between the point along the axis of the virtual element and thefocal point of the viewport exceeds the distance threshold; andpositioning the virtual element in the viewport such that the distancebetween the point along the axis of the virtual element and the focalpoint of the viewport is less than or equal to the distance threshold.33. The non-transitory computer-readable medium of claim 31, wherein theinstructions, when executed, further cause: determining that thedistance between the point along the axis of the virtual element and thefocal point of the viewport is less than or equal to the distancethreshold; and positioning the virtual element in the viewport such thatthe distance between the point along the axis of the virtual element andthe focal point of the viewport remains less than or equal to thedistance threshold.
 34. The non-transitory computer-readable medium ofclaim 31, wherein the instructions, when executed, further cause:determining that the distance between the point along the axis of thevirtual element and the focal point of the viewport is less than orequal to the distance threshold; and maintaining the positioning of thevirtual element in the viewport during the movement associated with theviewport.
 35. The non-transitory computer-readable medium of claim 29,wherein the movement associated with the viewport causes an oppositemovement of the virtual element in the viewport.
 36. The non-transitorycomputer-readable medium of claim 29, wherein the movement associatedwith the viewport comprises a predicted movement of at least a portionof a scene in the viewport and the point along the axis of the virtualelement comprises a predicted offset focal point based on the predictedmovement of the at least a portion of the scene in the viewport.
 37. Thenon-transitory computer-readable medium of claim 29, wherein the virtualelement is at least one of vertically elongated or horizontallyelongated.
 38. The non-transitory computer-readable medium of claim 29,wherein the determining the movement associated with the viewport isbased on at least one of: a movement of a camera, a movement of acomputing device, a movement of the viewport, data from a gyroscope ofthe computing device, data from an accelerometer of the computingdevice, or data from a movement sensor of the computing device.
 39. Thenon-transitory computer-readable medium of claim 38, wherein thecomputing device comprises at least one of a mobile device, a smartphone, a tablet computer, a portable gaming device, a wearable computingdevice, a head-mounted display, smart glasses, a virtual retinaldisplay, or a contact lens configured to display computer-generatedelements.
 40. The non-transitory computer-readable medium of claim 29,wherein the axis of the virtual element comprises a longitudinal axis ofthe virtual element.
 41. The non-transitory computer-readable medium ofclaim 29, wherein the viewport comprises an augmented reality (AR)viewport.
 42. The non-transitory computer-readable medium of claim 29,wherein the instructions, when executed, further cause: indicating, by auser and based on the positioning the virtual element in the viewport, afeature of the virtual element.